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(54) Image correction apparatus for stereo camera 

(57) An image correction apparatus for a stereo 
camera which corrects optical positional deviations in 
images taken by the stereo camera, comprising: 

means for performing interpolation on mutual refer- 
encing positions in stereo matching of a pair of 



images taken by the stereo camera by using pixel- 
by-pixel positional deviations that have been deter- 
mined in advance and coordinate data around the 
referencing positions. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present invention relates to an image cor- 
rection apparatus for a stereo camera which corrects an 
optical positional deviations in images taken by the 
stereo camera. 

2. Description of the Related Art : 

[0002] Image processing according to what is called a 
stereo method is known as a three-dimensional meas- 
uring technique using images. In this technique, in gen- 
eral, correlation between a pair of images obtained by 
imaging an object by two cameras (stereo camera) from 
different positions is determined and the distance is 
determined based on a parallax of the same object 
according to the principle of triangulation by using cam- 
era parameters such as installation positions and a focal 
length of the stereo camera. 

[0003] In this image processing according to the 
stereo method, a position where two image signals that 
are output from the stereo camera coincide with each 
other by superimposing the two image signals while 
shifting those sequentially. Therefore, essentially, it is 
desirable that there exist, between the two images, only 
deviations between corresponding positions as caused 
by a parallax; if there exist positional deviations due to 
other optical cfstortions etc., mismatching occurs and 
the accuracy of distance information is lowered. 
[0004] For the above reason, adjustments of optical 
positions are extremely important in using a stereo cam- 
era. Japanese Unexamined Patent Publication No. Hei. 
5-157557 discloses a technique of mechanically adjust- 
ing and holding correlation between two video cameras 
by providing, on a holding member for connecting and 
holding the two video cameras, a parallelism adjusting 
means for making adjustments so that the pixel array of 
an image sensor of one video camera becomes parallel 
with that of an image sensor of the other video camera 
and an optical axis adjusting member for making adjust- 
ments so that the optical axis of one video camera 
becomes parallel with that of the other video camera. 
[0005] However, conventionally, once deviations due 
to deterioration with age occur in a fixed stereo camera, 
readjustments should be performed in terms of 
mechanical structure. Readjustments not only require 
cumbersome operations but also take long time. Fur- 
ther, mechanical adjustments can assure only a limited 
degree of accuracy. 

[0006] To cope with the above problems, the present 
applicant has proposed, in Japanese Patent Application 
No. Hei. 9-117268, a technique of making adjustments 
electrically rather than mechanically by affine-trans- 
fbrming images in accordance with deviations in optical 



positions of a stereo camera. This techniques makes it 
possible to adjust optical positions of the stereo camera 
precisely to such a level as is difficult to attain by 
mechanical adjustments, and to easily perform read- 
5 justments on deviations due to deterioration with age 
after original adjustments. 

[0007] However, the correction technique by affine- 
transforming images that has been proposed by the 
present applicant is linear corrections, such as optical 

10 position corrections in the horizontal and vertical direc- 
tions and the rotational direction and a correction of a 
variation in the focal length (angle of view) of a lens, that 
act uniformly on the entire image. Therefore, this correc- 
tion technique cannot deal with nonlinear distortions in 

is an image due to influence of a lens distortion, Differ- 
ences in enlargement and reduction factors that depend 
on the region in an image as caused by inclination of the 
photodetecting surface of an imaging device, and other 
factors. 

20 [0008] Further, in a stereo camera, there may occur a 
case that to orient a distance information generation 
region to an optimum direction in accordance with use, 
a main camera for taking a reference image for stereo 
processing and a sub-camera for taking a comparison 
25 image for the stereo processing are mounted in 
advance so as to be inclined from the base line of the 
stereo camera. For example, in a vehicular stereo cam- 
era, there may occur a case that to locate a distance 
detection region evenly in the right-left direction with 
30 respect to the visual field in the right-left direction, the 
main camera and the sub-camera are so mounted as to 
be directed rightward with respect to the front side of the 
camera system (i.e., the front side of the self vehicle) 
and a sub-camera-side region, accounting for a pre- 
ss scribed proportion, of an image taken by the main cam- 
era is employed as a distance information generation 
region. 

[0009] Such inclined mounting of the two cameras 
with respect to the base line of the stereo camera 

40 requires nonlinear shape corrections on images taken. 
Corrections of nonlinear positional deviations in images 
including distortions of the above kinds due to a lens 
distortion, inclination of the photodetecting surface of an 
imaging device, etc. are an issue that remains to be 

45 addressed. 

ftl 1MMARY OF THE INVENTION 

[0010] The present invention has been made in view 
so of the above circumstances, and an object of the inven- 
tion is therefore to provide an image correction appara- 
tus for a stereo camera which can improve the reliability 
of stereo processing by making it possible to correct, in 
an on-board state, nonlinear positional deviations 
55 among optical positional deviations in images taken by 
the stereo camera. 

[0011] The above-described object of the invention 
can be attained by an image correction apparatus for a 
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stereo camera which corrects optical positional devia- 
tions in images taken by the stereo camera, comprising 
means for performing interpolation on mutual referenc- 
ing positions in stereo matching of a pair of images 
taken by the stereo camera by using pixel-by-pixel posi- 
tional deviations that have been determined in advance 
and coordinate data around the referencing positions. 
[0012] That is, in the invention, interpolation is per- 
formed on referencing positions in the counterpart 
image in stereo matching of a pair of images taken by 
the stereo camera by using pixel-by-pixel positional 
deviations that have been determined in advance and 
coordinate data around the referencing positions, and 
matching is made by referring to interpolated coordi- 
nates, whereby mismatching between corresponding 
positions can be prevented. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0013] 

Fig. 1 is a diagram showing the entire configuration 

of a stereo processing system. 

Fig. 2 is a diagram illustrating a correction pattern of 

a vertical/horizontal correction. 

Fig. 3 is a diagram illustrating a correction pattern of 

a rotational correction. 

Fig. 4 is a diagram illustrating a correction pattern of 
a focal length correction. 

Fig. 5 is a diagram illustrating a lens distortion cor- 
rection pattern. 

Fig. 6 is a diagram illustrating data interpolation in 
the vertical direction. 

Fig. 7 is a diagram illustrating a picked-up image of 

a horizontal lattice pattern. 

Fig. 8 is a diagram schematically illustrating a data 

distribution example of a coordinate correction 

table. 

Fig. 9 is a diagram illustrating an image obtained by 
performing a coordinate correction on the original 
image. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

[001 4] An embodiment of the present invention will be 
hereinafter described with reference to the drawings. 
Figs. 1 to 9 relate to the embodiment of the invention, in 
which Fig. 1 shows the entire configuration of a stereo 
processing system, Fig. 2 illustrates a correction pattern 
of a vertical/horizontal correction, Fig. 3 illustrates a cor- 
rection pattern of a rotational correction, Fig. 4 illus- 
trates a correction pattern of a focal length correction. 
Fig. 5 illustrates a lens distortion correction pattern, Fig. 
6 illustrates data interpolation in the vertical direction, 
Fig. 7 illustrates a picked-up image of a horizontal lattice 
pattern, Fig. 8 schematically illustrates a data distribu- 
tion example of a coordinate correction table, and Fig. 9 
illustrates an image obtained by performing a coordi- 



nate correction on the original image. 
[0015] Fig. 1 shows the configuration of a stereo 
image processing system which processes images 
taken by a stereo camera 1 that consists of two cameras 

5 each incorporating an image sensor such as a charge 
coupled device (CCD) and being variable in shutter 
speed. For example, this stereo image processing sys- 
tem is incorporated in a mobile body such as an auto- 
mobile or a helicopter and used as part of a system for 

10 recognizing its environment and its own position. 

[0016] One camera of the stereo camera 1 is a main 
camera for taking a reference image for stereo process- 
ing and the other camera is a sub-camera for taking a 
comparison image for the stereo processing. The stereo 

15 camera 1 is mounted on a camera stay (not shown) with 
a predetermined base line length. Two images taken by 
the respective cameras 2 and 3 are input via an image 
correction apparatus 10 as an image input apparatus to 
a stereo processing device 40, where the degree of 

20 coincidence of the two images is evaluated by stereo 
matching and a distance image is generated as three- 
dimensional image information in which far and near 
information is digitized. 

[001 7] In this case, although the two cameras 2 and 3 
25 are fixed to the camera stay after they were adjusted 
mechanically in an initial manufacturing stage so that 
their optical axes became parallel with each other, each 
of images taken by the two cameras 2 and 3 is not nec- 
essarily an optimum one for stereo processing because 
30 it includes considerable levels of optically nonlinear 
positional deviations due to limitations in mechanical 
assembling accuracy, influence of a variation in lens 
focal length and a lens distortion, differences in enlarge- 
ment and reduction factors that depend on the region in 
35 the picked-up image as caused by inclination of the pho- 
todetecting surface of an imaging device, and other fac- 
tors. 

[0018] In view of the above, the image correction 
apparatus 10 is so constructed as to provide optimum 

40 image data for stereo processing by correcting, on a 
pixel-by-pixel basis, optical positional deviations in an 
image taken by each camera 2 or 3. Analog images 
coming from the cameras 2 and 3 are converted by A/D 
converters 11 and 12 to digital images, which are then 

45 stored in an image memory 14 via a data bus switching 
circuit 13. The image data stored in the image memory 
14 are corrected by a data interpolation circuit 21 on a 
pixel-by-pixel basis. 

[0019] A reference/comparison image write address 
so generation circuit 16 and a reference/comparison image 
read address generation circuit 1 7 are connected to the 
image memory 14 via an address switching circuit 15. 
The reference/comparison image read address genera- 
tion circuit 17 accesses a reference/comparison image 
55 coordinate correction table 18 that stores pixel-by-pixel 
coordinate correction data. 

[0020] Coordinate correction data that is output from 
a coordinate correction value calculation circuit 19 for 
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calculating an image coordinate correction value in 
accordance with instructions from a microcomputer 50 
is added to a horizontal address that is output from the 
reference/comparison image read address generation 
circuit 17, and a resulting address is output to the 
address switching circuit 15. Coordinate correction data 
that is output from the reference/comparison image 
coordinate correction table 1 8 and coordinate correction 
data that is output from the coordinate correction value 
calculation circuit 19 are added to a vertical address 
that is output from the reference/comparison image 
read address generation circuit 17, and a resulting 
address is output to the data interpolation circuit 21 as 
well as to the address switching circuit 15 via a 2-step 
access circuit 20. 

[0021] An output of the data interpolation circuit 21 is 
input, together with an output of a shading correction 
table 22 that is accessed by the reference/comparison 
image read address generation circuit 17, to a lumi- 
nance correction result table 23, where it is subjected to 
a luminance correction. Luminance-corrected image 
data are output to a reference image memory 21 and a 
comparison image memory 32. 
[0022] Image data stored in the reference image 
memory 31 and the comparison image memory 32 are 
read into the stereo processing device 40 that is com- 
posed of a city block distance calculation circuit having 
a pipeline structure in which absolute value calculators 
and adders are connected together in pyramid form, a 
minimum value/pixel deviation detection circuit for eval- 
uating the minimum value, the maximum value, etc. of 
city block distances, and other circuits. In the stereo 
processing device 40, the degree of coincidence is eval- 
uated for each small region (e.g.. a small region of 4 x 4 
pixels) of the counterpart image and a distance image is 
generated. A distance image generation process is 
described in detail in Japanese Unexamined Patent 
Publication No. Hei. 5-1 14099 of the present applicant. 
[0023] Further, distance image data that is output from 
the stereo processing device 40 and corrected original 
image data that is output from the image correction 
apparatus 10 are read into the microcomputer 50, 
where various recognition processes are executed. For 
example, when the system is applied to a vehicle, the 
microcomputer 50 recognizes a running environment 
such as a road shape and presence of a vehicle running 
ahead and outputs vehicle control data for, for example, 
alarming the driver by judging the degree of danger of 
collision, stopping the vehicle by activating the brake 
automatically, or increasing or decreasing the running 
speed automatically to maintain a safe distance from 
the vehicle running ahead. 

[0024] In the above stereo processing system, a refer- 
ence image taken by the main camera 2 and a compar- 
ison image taken by the sub-camera 3 are processed in 
a time-divisional manner by the image correction appa- 
ratus 10 as an image input apparatus. Image data of a 
small region for stereo matching is stored in the refer- 



ence image memory 31. Image data for referring to a 
position corresponding to one small region is stored in 
the comparison image memory 32 by an amount cover- 
ing a prescribed number of pixels of deviation in the hor- 
5 izontal direction (for example, an amount covering 100 
pixels of deviation for a small region of 4 x 4 pixels, i.e., 
data of 4x104 pixels). 

[0025] That is, in the image correction apparatus 10, 
analog images taken by the respective cameras 2 and 3 

w are converted by the A/D converters 1 1 and 1 2 to digital 
images of prescribed luminance gradation levels (e.g., 
gray scale of 256 gradation levels) and digital image 
data are stored in the image memory 14 at horizontal 
and vertical addresses specified by the reference/com- 

15 parison image write address generation circuit 16 with 
the bus switched to the image memory 14 side by the 
data bus switching circuit 13. 

[0026] When prescribed image data are stored in the 
image memory 14, the reference/comparison image 

20 read address generation circuit 17 starts image data 
reading. That is, the reference/comparison image coor- 
dinate correction table 18 is accessed and coordinate 
correction data (address correction data) is added to a 
vertical address, and coordinate correction data that is 

25 output from the coordinate correction value calculation 
circuit 19 is added to a horizontal address and the verti- 
cal address. The image memory 14 is accessed via the 
address switching circuit 15. 

[0027] In this embodiment, the coordinate correction 

30 data that are stored in the reference/comparison image 
coordinate correction table 18 are data to be used for 
correcting components that are perpendicular to the 
scanning direction (horizontal direction) of the stereo 
matching, that is, initial correction data for correcting 

35 top-bottom deviations, rotational deviations, a lens dis- 
tortion, and a focal length variation in reference and 
comparison images. For example, the coordinate cor- 
rection data are 8-bit data in which the highest one bit 
represents a positive/negative sign indicating a correc- 

40 tion direction, the next four bits represent an integer por- 
tion, and the lowest 3 bits represent a decimal portion. 
In this case, the correction is such that the maximum 
coordinate correction value is ±16 pixels and the mini- 
mum coordinate correction resolution is 0.125 (1/8) 

45 pixel. 

[0028] The reference/comparison image coordinate 
correction table 18 is generated in advance by imaging 
a particular imaging pattern for correction value meas- 
urement such as a square lattice pattern with the two 

so cameras 2 and 3 at the same time and calculating coor- 
dinate correction values on a pixel-by-pixel basis for the 
images taken by the cameras 2 and 3. That is, the coor- 
dinates of the center of each lattice point is measured 
from an image taken with less than one pixel as a unit. 

55 a deviation of each lattice point is calculated by compar- 
ing the measured coordinates with a corresponding lat- 
tice point position of a reference square lattice pattern 
stored in a measuring computer, and a correction value 
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for each lattice point is determined by subjecting the 
deviation to a data form conversion in which the correc- 
tion direction is taken into consideration. A coordinate 
correction amount of each pixel is calculated by 4-point 
interpolation based on correction values of four lattice 5 
points around the pixel concerned, and stored in the 
table 18. 

[0029] If appropriate, the reference/comparison image 
coordinate correction table 1 8 may store correction data 
only in the vertical or horizontal direction. Figs. 7-9 10 
shows an example in which correction data in the verti- 
cal direction are stored in the table 18. In this example, 
a horizontal lattice pattern for correction value measure- 
ment is imaged in advance by the two cameras 2 and 3 
at the same time. Fig. 7 shows an example image taken is 
by the camera 2 or 3. A center line (center-of-gravity 
line) located at the center of the width of each horizontal 
lattice line is determined from the image taken, and 
each center-of-gravity line is compared with a reference 
horizontal lattice that is stored in a measuring computer. 20 
Vertical coordinate correction values at respective pix- 
els on the reference lattice are determined, and a coor- 
dinate correction table having a data distribution as 
shown in Fig. 8 is generated by determining coordinate 
correction values of pixels that are not located on the 25 
reference lattice by linear interpolation. 
[0030] Fig. 9 shows a result of a correction in which 
the image of Fig. 7 that is obtained by imaging the hori- 
zontal lattice pattern has been corrected by using the 
coordinate correction table of Rg. 8. It is seen that 30 
remarkable distortions at four corner portions of the 
image taken are corrected and all the lattice lines are 
thereby corrected so as to extend horizontally. 
[0031] Where correction data in the horizontal direc- 
tion are stored in the table 18, a process similar to the 35 
above is executed by using a vertical lattice pattern. 
Where correction data in the vertical or horizontal direc- 
tion are stored in the table 18, correction data in the 
other direction may be calculated by the coordinate cor- 
rection value calculation circuit 1 9 (described later). 40 
[0032] On the other hand, coordinate correction data 
that are output from the coordinate correction value cal- 
culation circuit 19 are data to be used for correcting top- 
bottom deviations, right-left deviations, rotational devia- 
tions, etc. of the cameras 2 and 3 due to deterioration 45 
with age based on instructions from the microcomputer 
50, and are correction data that act on the entire image 
uniformly. In this correction, while a vertical correction is 
performed down to a 1/8 pixel, a vertical/horizontal 
coordinate correction is performed with one pixel as a so 
unit. In this case, a coordinate correction amount in the 
horizontal direction is so set that corresponding posi- 
tions in the counterpart image deviate by a predeter- 
mined number of pixels (e.g.. 3 pixels) at infinity so that 
the entire object to infinity is included in the scanning ss 
range of the stereo matching. A simple version of a cor- 
rection of rotational deviations can be realized by 
increasing or decreasing the vertical correction amount 



as the horizontal address increases, in accordance with 
rotational deviation correction data that are set by the 
microcomputer. 

[0033] For example, as for a vertical/horizontal correc- 
tion on top-bottom deviations or right-left deviations as 
shown in Fig. 2, correction is made at a minimum of a 
1/8-pixel pitch in the vertical direction and 1 -pixel pitch 
in the horizontal direction. As for a correction on rota- 
tional deviations as shown in Fig. 3, a correction on a 
focal length variation as shown in Fig. 4, and a correc- 
tion on a lens distortion as shown in Fig. 5, correction is 
made at a minimum of a 1/8 pixel pitch only in the verti- 
cal direction. 

[0034] That is, where stereo matching is performed in 
a small area having a relatively small number of pixels 
as in the case of this embodiment, data interpolation on 
only components that are perpendicular to the scanning 
direction of the stereo matching is performed at a reso- 
lution of less than one pixel and read addresses are 
shifted in the horizontal direction with one pixel as a 
unit, because information of luminance differences 
between adjacent pixels is lost in the data interpolation 
though the information loss is small, the processing time 
is long, and so forth. 

[0035] Therefore, as for data that is obtained by coor- 
dinate-correcting a vertical address generated by the 
reference/comparison read address generation circuit 
17, the lower 3 bits representing a decimal portion are 
output to the data interpolation circuit 21 and the upper 
5 bits are used by the 2-step access circuit 20 to 
access, in two steps, two pixels located above and 
below in the vertical direction. Based on those data, the 
data interpolation circuit 21 performs interpolation on 
data of mutual referencing positions of the reference 
image and the comparison image in the stereo match- 
ing. 

[0036] When coordinates A(x, y) of a certain pixel of 
the original image in the image memory 14 is deviated 
by m pixels from its proper position as shown in Fig. 6. 
the data interpolation circuit 21 determines positional- 
deviation-corrected coordinates B(x, y) by performing 
interpolation according to Equation (1) below in the 
case of a positive deviation or Equation (2) below in the 
case of a negative deviation by using coordinates b(x, y- 
1) and b(x, y+1) of two pixels located above and below 
in the vertical direction that are obtained through access 
by the 2-step access circuit 20. This process is exe- 
cuted for all pixels of the original image. 

B(x, y) *= b(x, y) x (1 - m) + b(x, y-1) x m (1) 

B(x, y) = b(x, y) x x (1 - m) + b(x, y+1) x m (2) 

[0037] In this case, where corrections in both of the 
vertical and horizontal directions are performed with 
less than one pixel as a unit, coordinate data around a 
referencing position is determined by making 2-step 
access for each of a horizontal address and a vertical 
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address and vertical interpolation and horizontal inter- 
polation are performed by using those coordinate data 
and coordinate data obtained from the horizontal 
address line and the vertical address line. 
[0038] In the luminance correction result table 23, 
image data corrected by the above-described data 
interpolation circuit 21 are corrected for a luminance 
reduction due to a shading phenomenon that occurs in 
the optical system of each camera by using data that 
are output from the shading correction table 22 that is 
accessed by the reference/comparison image read 
address generation circuit 1 7. Resulting data are stored 
in the reference image memory 31 and the comparison 
image memory 32. 

[0039] The image data stored in the reference image 
memory 31 and the comparison image memory 32 are 
read into the stereo processing device 40 and subjected 
to stereo matching there In the stereo matching, a proc- 
ess of calculating a city block distance between one 
small region of the reference image memory 31 and that 
of the comparison image memory 32 is repeated while 
a shift is made in the horizontal direction by one pixel 
each time until a shift of a predetermined number of pix- 
els is made. The minimum value, the maximum value, 
etc. of city block distances are evaluated and it is 
checked whether the minimum value of city block dis- 
tances truly indicates coincidence of the small regions 
of the two images. If check conditions are satisfied, a 
pixel deviation amount at which the city block distance is 
minimized is output as distance information correspond- 
ing to the small region of the reference image. 
[0040] The stereo matching is performed in a state 
that not only influence of linear positional deviations that 
occur uniformly in the entire image taken by each cam- 
era 2 or 3 but also influence of nonlinear positional devi- 
ations for which the correction value varies locally have 
been removed in an on-board state. Therefore, there 
occurs no mismatching between corresponding posi- 
tions and hence correct distance information is 
obtained, which makes it possible to improve the relia- 
bility of image processing. 

[0041] Further, even where the two cameras 2 and 3 
are mounted so as to be inclined from the base line, 
nonlinear shape corrections can be performed on an 
image taken from an oblique direction. A flexible system 
can thus be realized. 

[0042] As described above, according to the invention, 
interpolation is performed on referencing positions in 
the counterpart image in stereo matching of a pair of 
images taken by the stereo camera by using pixel-by- 
pixel positional deviations that have been determined in 
advance and coordinate data around the referencing 
positions. Therefore, not only influence of linear posi- 
tional deviations that occur uniformly in the entire image 
taken by each camera 2 or 3 but also influence of non- 
linear positional deviations for which the correction 
value varies locally have been removed in an on-board 
state. Therefore, there occurs no mismatching between 



corresponding positions and hence correct distance 
information is obtained, which makes it possible to 
improve the reliability of image processing. Further, 
even where the stereo camera is mounted so as to be 
5 inclined from the base line, nonlinear shape corrections 
can be performed on an image taken from an oblique 
direction. Great advantages can thus be obtained as 
exemplified by an advantage that a flexible system can 
be realized. 

10 

Claims 

1 . An image correction apparatus for a stereo camera 
which corrects optical positional deviations in 
is images taken by the stereo camera, comprising: 

means for performing interpolation on mutual 
referencing positions in stereo matching of a 
pair of images taken by the stereo camera by 
20 using pixel-by-pixel positional deviations that 

have been determined in advance and coordi- 
nate data around the referencing positions. 
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